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Objective 

The objective of this research was to explore design thinking among teams of high school students. 
This objective is encompassed in the research question driving this inquiry: How do teams of high school 
students allocate time across stages of design? Design thinking on the professional level typically occurs 
in a team environment. Many individuals contribute in a variety of ways to facilitate the successful 
development of a solution to a problem. Teachers often require students to work in groups, but little is 
known about how the group functions in the context of design and the potential interaction between group 
performance and authentic design challenges. Few research results are available to guide teachers in 
developing successful design teams and encouraging them in their efforts. 

Problem Statement 

The discrepancy between our society’s dependence upon technology and our ability to understand 
various technological issues has emerged as a serious concern for educators. “Technology is the outcome 
of engineering; it is rare that science translates directly into technology, just as it is not true that 
engineering is just applied science” (National Academy of Engineering, 2004, p. 7). Specifically, 
“Americans are poorly equipped to recognize, let alone ponder or address, the challenges technology 
poses or the problems it could solve” (Pearson & Young, 2002, pp. 1-2). The relationship between 
understanding engineering and technological literacy is of special urgency during the high school years, 
since “technologically literate people should also know something about the engineering design process” 
(Pearson & Young, 2002, p. 18). 

Design thinking is fundamental to understanding the technologically dependent nature of our society. 
A need for a technologically literate populace, therefore, includes an understanding of the engineering 
design process. The design process links technology and engineering, two elements of STEM education. 
“Design is the central component of the practice of engineering and a key element in technology 
education” (Pearson & Young, 2002, p. 58). This study identified quality high school technology and 
engineering learning and teaching environments in a criterion based sampling strategy, the setting 
envisioned by Pearson and Y oung, where “technology teachers with a good understanding of science and 
the interactions between technology, science, and society will be well prepared to work with other 
teachers to integrate technology with other subjects” (2002, p. 108). 

While design thinking is an elusive and difficult construct to define, measurements for this study 
included a pertinent subset of measurements consistent with previous literature, much of which was 
generated through work of the Center for Engineering Learning and Teaching (Atman, Chimka, Bursic, & 
Nachtmann, 1999; Atman, Kilgore, & McKenna, 2008; Morozov, Yasuhara, Kilgore, & Atman, 2008; 
Mosborg et al., 2005; Mosborg et al., 2006). This paper reports on measurements including time allocated 
across essential elements of the design process elements of the design process and disaggregates the data 
by problem type and the gender composition of the teams. 
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Engineering Design Problems 


According to the National Center for Engineering and Technology Education Caucus Report of 2012, 
“There is a need for more definitive guidance about what makes quality design challenges and how they 
can be implemented well in existing courses” (Householder & Hailey, 2012, p. 2). Two different design 
problems were administered in this study. The “playground problem” was comparable to the design 
problem used in previous studies with individual high school students (Mentzer, Becker, & Park, 2011) 
and previous work with college students and experts (Atman, et al., 1999; Mosborg, et al., 2005; 
Mosborg, et al., 2006). The “pedestrian flow problem” administered to approximately half of the teams 
was a variation of the “street crossing problem” adapted from previous literature (Cardella, Atman, Turns, 
& Adams, 2008; Mullins, Atman, & Shuman, 1999). The “playground problem” was provided to pemiit 
comparisons between team and individual performances while the “street crossing problem” variation 
facilitated comparisons among types of problem structures. The street crossing problem was less 
structured, could be readily adapted in order to be locally relevant, and was potentially more authentic for 
the participants, who also had an opportunity to specify constraints and criteria for their problem. 

The Playground Problem has been used in multiple studies and can be traced to Dally and Zang 
(1993). The original need for project driven approaches in the freshman engineering design course was to 
increase student performance and retention and to situate student learning of abstract concepts through 
real world applications in an experiential activity. In the original activity, students designed a swing set 
with slides and seesaw. Atman et al. (1999) revised the foundational work of Dally and Zang to create a 
playground design problem. In their challenge, engineering students were presented with a brief 
playground design task and access to background information upon request. Participants were provided 
with a maximum of three hours to develop a solution to the problem while thinking aloud. Mosborg et al. 
(2005) applied the playground design challenge using the “think aloud” research protocol with 19 
practicing engineers who were identified as experts in the field. Mosborg et al. (2006) compared groups 
of freshman and senior engineering students with practicing engineers using data previously collected on 
the playground design challenge. Atman et al. (2008) analyzed data from previous studies with a focus on 
the language of design, its relationship to design thinking as a mediator, and relationships between the 
internalization of design thinking and language acquisition. The endeavor to model problem solving 
satisfactorily has engaged scholars across domains (Hayes, 1989; Newell & Simon, 1972; Polya, 1945; 
Rubenzer, 1979). The playground design task, an open-ended, realistic, accessible, and complex problem, 
is an effective design challenge to enable researchers to study design thinking (Mosborg et al., 2006). 

In the previous studies, participants were given a one page design brief of the playground problem. 
The participating teams, acting as engineers, were assigned to design a playground on a donated city 
block. The constraints include limited budget, child safety, and compliance with zoning regulations and 
applicable laws. Participants were able to query the research administrator for additional specific 
information on the lot layout, cost of materials, neighborhood demographics, or other information. There 
was a two-hour time limit for completion of the design proposal, which was a modification of the original 
three hour limit. This modification was made because the average design time in the previous study of 
high school students was about 90 minutes. The two-hour limit provided more time than the average 
individual needed, yet it reduced the resources needed for data collection. The participants presented a 
written proposal describing their design. This activity engaged the participants in problem framing and 
the development of an initial solution. Limitations of this design task included the lack of opportunity for 
participants to investigate the need for a solution, since the problem was simply assigned to them. 
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Students did not have an opportunity to construct physical models or prototypes. Participants were aware 
that implementation of the design project would not occur, that their designs would not be realized. 

The Street Crossing Design Problem was adapted from previous research (Cardella, et al., 2008; 
Mullins, et al., 1999). The National Center for Engineering and Technology Education assembled a 
Caucus in August 201 1 to identify characteristics of engineering design challenges (Householder, 201 1). 
Results of the discussions by this group of experts indicated that excellent design challenges should 
incorporate the following characteristics: 

• Authenticity 

• Have personal and social relevance 

• Require analytical thinking 

• Involve group efforts 

• Require hands-on participation 

• Are clearly structured but open-ended 

• Foster creative solutions 

• Consider ethical issues 

• Meet applicable constraints 

• Provide opportunities for modeling with replication 

• Consider systems implications 

• Are well documented 

• Are self-assessed and independently evaluated 

• Enable communication among team members 

The Street Crossing Design Problem was potentially more authentic and more closely aligned with 
the National Center’s Caucus suggestions. “Authentic problems currently affect real-life situations 
encountered by the learners, their families, and their communities - and they do not have a generally 
recognized “right answer.” (Householder & Hailey, 2012, p. 22). The problem was modified slightly from 
its original administration to more closely exemplify these characteristics. This modification presented a 
unique opportunity to contrast results on the street crossing design problem with results on the playground 
problem, which has been used extensively, but does not meet these characteristics as well for the target 
audience. 

The street crossing problem was presented to teachers at the schools involved. Teachers were asked to 
think about an intersection fitting characteristics of the original problem but located at or near the school 
where students would immediately recognize the problem as they personally experienced it daily. After 
negotiating with the teachers, it was discovered that car/pedestrian traffic flow was an issue, but a more 
relevant and pervasive similar issue was pedestrian (student) flow in school hallways. Each school had 
one or more significant blockages that caused congestion, frustration and delay at passing times between 
classes. The problem was modified to focus on student hallway flow rather than traffic flow, making the 
problem more relevant and personal, as most students experience the congestion several times per day. 

Each pedestrian flow problem was presented in a similar format: the school floor plan (“map”) was 
provided to participants along with a brief narrative stating that the student team was a team of engineers 
contacted by the school district. The narrative introduced an area that the students immediately 
recognized as a congested area and requested that the team present a proposal for resolving the issue. The 
constraints and criteria were not specified; leaving student design teams the opportunity to discuss and 
negotiate their specific problem definition and determine the most appropriate solution proposal. This less 
structured problem is consistent with the National Center’s suggestion that “Engineering design 
challenges are ill-structured problems that may be approached and resolved using strategies and 
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approaches commonly considered to be engineering practices” (Householder & Hailey, 2012, p. 2). 
Typical office supplies were provided for the participants, a condition similar to those in the playground 
problem. However, participants were given ready access to the Internet in lieu of printed sheets of 
relevant data. The decision to provide Internet access was made to increase the sense of authenticity and 
relevance as students are familiar with and accustomed to having Internet access. The notion of having 
predetermined what information is needed for their solutions may unintentionally guide student design 
decisions to those based upon a finite resource pool. 

Methodology 

A descriptive study was conducted spanning multiple high schools in urban, rural and suburban 
environments. The quantitative research method design leveraged the use of data from 17 design teams 
comprised of 2-4 students each. Seven of the teams were comprised of males, five were comprised of 
females, and five teams included both male and female members. Seven of the teams received the 
playground design problem and eight received a locally relevant school hallway traffic flow problem. 
Teams were expected to develop a solution in 2 hours. The interactions of group members were video and 
audio recorded while they were developing the design solution. 

Sample. A sample of 47 students was used in this study. The teacher grouped the students into teams 
according to their personal schedules and the teams were assigned the design challenge. Some team 
members were friends and other teams were comprised of students who did not know each other well. 

The 17 teams that were created came from four schools in two states. Each of the schools selected to 
participate had a recognized engineering program associated with an outreach effort by a university 
engineering program. Curricular offerings at the high schools included Project Lead the Way (PLTW), 
Engineering Projects in Community Service (EPICS) High, First Robotics, and locally developed 
engineering and/or technology courses supported by their regional University. 

Teachers at the target schools permitted advertising to recruit their students for participation in the 
study. Students in this study were considered to be representative of experienced students who had taken 
most or all engineering related courses at their high school. Students were recruited who were actively 
engaged in the study of engineering design through a criterion sampling strategy (Creswell, 1998) using 
the following criteria: 

• The high schools had an established program of study which employs a focus on engineering in a 
sequence of courses developed in association with an engineering outreach effort as part of a 
university program. 

• In these courses, students participated in design activities which engage their critical thinking and 
problem solving skills within the framework of the engineering design process. 

• Students were selected who represent diverse backgrounds and have chosen to enroll in this 
sequence of courses. 

Administration of the Design Challenges. A team of graduate and undergraduate students and a 
university faculty member conducted the data collection. A lead researcher administered the problem and 
trained the student researchers through discussion, observation and direct participation. The researcher 
reviewed data collected and reflected with the students following the session to standardize procedures 
and provide oversight on consistent research administration. Student researchers’ administration of the 
design problem was video recorded as part of the data collection and videos were reviewed for training 
purposes to ensure consistency during data collection. 

Data included video and audio recordings of the design sessions. Video cameras were small, mounted 
on miniature tripods to minimize their intrusion. Audio recorders were used as a backup to the video 
cameras and were positioned on the work space near the student. All students were wired with a lavalier 
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microphone to ensure high quality audio feeds. Wires were run under the team workspace to prevent 
tangling, however, the wires limited student mobility. Students generated documents and other artifacts 
with traditional office supplies provided. Artifacts typically included sketches, notes, and formal 
drawings. Two-dimensional works were anticipated by the research team and scanned to digital image 
form. Data were archived in digital format on hard drives. Drives were shared with the research team for 
analysis purposes. 

Data Analysis. The playground problem coding scheme was congruent with the approach used in 
earlier studies (Atman, et al., 1999; Bursic & Atman, 1997; Mosborg, et al., 2005; Mosborg, et al., 2006). 
Time is a limited resource and the ways designers allocate their time among the areas of the design 
process has been a focus of previous work. Two measurements of time were made while the designers are 
at work: time allocated to elements of the design process; and total time engaged in design. The unit of 
analysis was the team. The coding team used NVIVO software to analyze each video. The coding scheme 
was similar to the approach used in prior studies (Atman, et al., 1999; Mosborg, et al., 2005; Mosborg, et 
al., 2006). The data were coded into the nine categories presented by Mosborg et al. (2006, p. 15): (1) 
Problem Definition, defining what the problem really is; (2) Gather Information, searching for and 
collecting information needed to solve the problem; (3) Generating Ideas, thinking up potential solutions 
(or parts of potential solution) to the problem; (4) Modeling, detailing how to build the solution (or parts 
of the solution) to the problem; (5) Feasibility Analysis, assessing and passing judgment on a possible or 
planned solution to the problem; (6) Evaluation, comparing and contrasting two (or more) solutions to the 
problem on a particular dimension (or set of dimensions) such as strength or cost; (7) Decision, selecting 
one idea or solution to the problem (or parts of the problem) from among those considered; (8) 
Communication, the participants’ communicating elements of the design in writing, or with oral reports, 
to parties such as contractors and the community; and (9) Other, none of the above codes apply. 

Data analysis began with segmenting the data sets. A team of three researchers were tasked with the 
responsibility of segmenting. A segment was defined as a pause bound utterance, as suggested by Atman 
et al. (1999). Researchers were instructed to create a new segment in the video timeline for each instance 
when any student on the team began a new thought, which was typically indicated by beginning to speak 
after a pause. In previous literature, this segmenting procedure was applied to individuals. For the current 
study of teams, researchers created these segments each time any member of the team made a transition. 
The resulting segmented data represented the composite of all team member segments. At some points in 
the videos, all team members were functioning as one cohesive unit and segmenting was simple and a 
single layer. In other times, a team of four students might naturally divide into two teams of two and the 
segments represent start/stop times for each sub-team. By segmenting in this fashion, a divergence in 
design activities could be coded with two separate codes with two different, but overlapping episodes. 

Quantitative measures of inter rater reliability on segmenting were not made. The research leadership 
determined the segmenting would be of reasonable quality if the inter -rater reliability measures for coding 
were high. If segmenting were done successfully, coding could potentially result in high inter rater 
reliability. Coding served as a proxy for quality control of the segmenting process. As a preliminary 
quality control mechanism, the lead researcher reviewed segmented work and provided feedback and 
guidance as the research assistants progressed. 

Two undergraduate students coded the data. Three phases were conducted. The first was to establish 
calibration of the research assistant’s coding work. The second phase served to document the calibration 
using Kappa values as a measure of inter-rater reliability. The team of two research assistants coded 25% 
of each video and compared. The Kappa values averaged 0.71; details are presented in Table 1 along with 
the number of references used to generate the values. In the third phase, all videos were fully coded, 
approximately one -half by one research assistant and one -half by the other. 
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In the calibration phase, research assistants were provided with a conceptual overview of the coding 
process, structure, technique and rationale. They were presented with examples from previous work and 
practiced coding these data. Research assistants then coded a portion of a video and compared with each 
other. They would meet with a senior research team member and discuss the individual interpretations 
and differences seeking clarification on coding. A “Dynamic” Code Book was adopted and maintained. 
This was a document with very specific examples of the different codes developed by creating a 
description of the code and compilation examples in context. This included adding detail and clarifying 
the meaning of our segmenting and coding procedures and providing examples as coders did their work. 
The document was updated regularly and shared via network real time. As understanding and 
interpretation was negotiated by the coders and research team leaders, the codebook documents the 
increasingly specific definitions. 

While the coding scheme was consistent with previous literature, the technique was slightly different. 
Previous work used transcription, segmenting and coding as three separate activities in the analysis 
process (Atman, et al., 1999). Inter-rater reliability was calculated on the coding to ensure reliability of 
the multiple coding analysts. Our project bypassed transcription by using NVIVO software which 
presented coding analysts with synchronized video and audio feed. Codes were associated with the 
timeline on the video/audio tracks and inter-rater reliability was computed using the Cohen’s Kappa 
statistic. 

The calibration process was iterative. Each coding session was followed by a debriefing session and 
the cycle started over. Kappa values began relatively low and increased gradually as the research 
assistants became more closely aligned in their designations. When average Kappa values for each code 
approached 0.70, the research team transitioned into the next phase, documentation. Some effort was 
focused on calibration, but most effort was allocated toward coding a random 25% of each video and 
documenting the comparison. 


Table 1 


CoheiTs Kappa For Each Design Activity 


Design Activity 

Cohen’s Kappa 

References 

Problem Definition 

0.76 

152 

Gathering Information 

0.72 

630 

Generating Ideas 

0.66 

65 

Modeling 

0.68 

1412 

Feasibility 

0.46 

294 

Evaluation 

0.75 

26 

Decision 

0.80 

15 

Communication 

0.88 

732 

Average Inter-Rater Reliability 

0.71 



In the final phase of coding, the 17 videos were divided among the two research assistants. Earlier 
work resulted in 25% of each video being coded already; the remaining 75% was coded. The entire video 
was reviewed and changes were made as needed to the coding structure in context of the newly coded 


75%. 
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Results 


Team time allocation was measured as a proxy of effort in the design process. Student teams were 
considered the units of analysis and the video provided data on team performance. The video data were 
coded by time allocated to: Problem Definition, Gathering Information, Generating Ideas, Modeling, 
Feasibility, Evaluation, Decision Making and Communication efforts. Activities that the team engaged in 
were coded. At times in the process, team members were all simultaneously engaged in one activity, but 
at other times, individual students would engage in different activities. When team members provided 
reasonable evidence that they were doing two different activities, two or more codes were applied. The 
total coded data exceeded 100% in all teams because, at times, the team was receiving credit for two or 
more codes simultaneously. Teams averaged 102 minutes in the design process as compared to 
individuals from previous work who finished, on average, at 92 minutes (Becker, Mentzer, & Park, 2012). 
Table 2 shows the average time invested by the teams in each design activity in this study and the average 
time invested by individuals in the previous study. 


Table 2 

Mean and Standard Deviation Summary Statistics for High School Student Teams 
and Individuals 


Design Process Measures 

Individual 

(n=59) 

Teams 

(n=17) 



Minutes (SD) 

Percent 
of time 

Minutes (SD) 

Percent 
of time 

Total Time 

Problem Scoping stage 

91.7 (47.4) 

15.5 

18.0 

101.7(18.43) 

27.2 

26.3 

Problem Definition 

5.6 (3.1) 

7.7 

6.8 (5.49) 

6.7 

Gathering Information 

9.9(13.3) 

10.3 

20.4(12.19) 

19.7 

Developing Alternative 

63.2 

70.5 

55.6 

55.1 

Solutions stage 
Generating Ideas 

2. 9(6. 6) 

3.9 

2.8(1.62) 

2.9 

Modeling 

54.4 (35.4) 

60.2 

44.2 (13.29) 

43.4 

Feasibility Analysis 

4.4 (4.1) 

5.4 

8.0 (4.41) 

8.3 

Evaluation 

1.1 (3.5) 

1.0 

0.5 (0.78) 

0.5 

Project Realization stage 

8.2 

7.6 

24.2 

23.6 

Decision 

0.4 (0.7) 

0.4 

0.2 (0.32) 

0.2 

Communication 

7.8(13.0) 

7.2 

24.0 (12.87) 

23.4 

Other 

3.1 

3.8 

9.1 

9.0 


With this study’s small sample size (n=17) statistical analysis was not conducted. However, trends 
emergent in the time allocation between individuals and teams may provide a foundation for future study. 
Teams spent nearly twice the percentage of time engaged in gathering information. Information gathering 
was coded when students were actively requesting, reading, and reviewing information related to the 
problem or solution. Information requests could be made of the administrator. Teams working on the 
hallway traffic design challenge were provided with a laptop and Internet access. Student use of the 
Internet was generally coded as information gathering and represents a difference from the data collection 
protocol used with individuals and playground design teams as they did not have access to the computer. 
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Modeling and communication time allocations show differences between teams and individuals. The 
teams tended to spend less time modeling and more time communicating. Modeling was defined as 
detailing how to build something, including calculations, estimations, determining locations, and 
description of how something will be assembled or fabricated. Communicating was defined as the efforts 
involved in telling someone how to build the playground. Communication efforts focused on sharing the 
team’s plan with others and could be directed toward a contractor or a board of directors considering the 
team’s proposal. 

Differentiating between modeling and communication in the abstract was simple for the 
undergraduate research assistants as the difference centered on purpose. If the purpose of the sketching, 
for example, was to understand and improve appearance, functionality or fabrication techniques and the 
team used this information to think through challenges and determine specifications, it was coded as 
modeling. If, on the other hand, team effort was directed at documenting their plans for fabrication for the 
purpose of telling someone how to build from the plans, it was coded as communicating. Student teams 
often started modeling and the work evolved into communication. This evolution made precise 
determination difficult. In cases where the transition was gradual and vague, the coders generally 
defaulted to modeling until there was evidence that the purpose was an attempt to communicate team 
intentions/plans. In some cases, teams were very deliberate about this transition. In other cases, the 
transitions occurred gradually but were clarified later. For example, what might have appeared to be 
modeling was later determined to be communication and codes were changed appropriately as the coding 
process progressed through the team’s work. 

An underlying assumption in generalizing previous work (Becker, et al., 2012) to a larger population 
of high school students is that time allocated to elements of the design process is representative of student 
understanding of process and not overly dependent on the nature of the problem. In this study, two 
different problems were used. Table 2 (above) represents both problems in the team environment and 
compares to individuals who were engaged in just one of those problems. Table 3 compares teams who 
engaged in the playground problem (n=8) and the pedestrian flow problem (n=9). These small sample 
sizes preclude the use of meaningful statistical analysis, but do provide a hint into a potential discovery 
which could serve as a foundation for future work. Teams for each design problem included single gender 
and mixed gender compositions. The playground problem was administered to one female team, five male 
teams and two mixed gender teams. The pedestrian flow problem administrations included three female 
teams, three male teams and three mixed gender teams. 

Time spent on problem definition differed between the pedestrian flow and playground problems. 
Teams spent more than twice the amount of time reading, considering, reflecting on and considering the 
problem for the playground compared to the pedestrian flow challenge. The playground problem 
presentation was longer and more specific. Constraints and criteria were specified as compared to the 
pedestrian flow in which constraints and criteria were not specified. The research team had anticipated 
that the lack of definition would permit students to develop their own constraints and criteria relative to 
their local problem, but time in the problem definition phase was actually less when constraints and 
criteria were not provided. 

The following is an excerpt from the playground design problem as adopted from Atman (1999; 
2008): 

You estimate that most of the children who will use the playground will range from 1 to 10 years 
of age. Twelve children should be kept busy at any one time. There should be at least three 
different types of activities for the children. Any equipment you design must 

• be safe for the children 

• remain outside all year long 

• not cost too much 
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• comply with the Americans with Disabilities Act 
The neighborhood does not have the time or money to buy ready-made pieces of equipment. Your 
design should use materials that are available at any hardware or lumber store. The playground 
must be ready for use in 2 months. 


Table 3 

Mean and Standard Deviation Summary Statistics for High School Student Teams 
in the Playground Problem and Pedestrian Flow Problem 


Design Process Measures 

Playground Teams 
(n=8) 

Pedestrian Flow Teams 
(n=9) 


Minutes (SD) 

Percent 
of time 

Minutes (SD) 

Percent 
of time 

Total Time 

Problem Scoping stage 

108(10.9) 

25.3 

23.0 

95.8(21.5) 

28.9 

29.4 

Problem Definition 

9.9 (3.7) 

9.2 

4.1 (5.4) 

4.4 

Gathering Information 

15.4 (9.0) 

13.7 

24.8 (12.9) 

25.0 

Developing Alternative 

62.4 

57.8 

49.5 

52.6 

Solutions stage 
Generating Ideas 

2.7(1. 9) 

2.5 

2.9 (1.4) 

3.2 

Modeling 

53.2 (10.0) 

49.4 

36.1 (10.3) 

38.1 

Feasibility Analysis 

5.4 (1.7) 

4.9 

10.4 (4.7) 

11.3 

Evaluation 

1.1 (0.8) 

1.0 

<0.1 «0.1) 

<0.1 

Project Realization stage 

29.6 

27.9 

19.4 

19.8 

Decision 

0.5 (0.3) 

0.5 

<0.1 «0.1) 

<0.1 

Communication 

29.2(11.4) 

27.4 

19.4(12.4) 

19.8 

Other 

9.7 

9.1 

8.5 

9.2 


In this exceipt. constraints included the target audience of users, the number of children using the 
playground simultaneously, the number of different types of pieces of the playground that will remain 
outside, and a timeline. Criteria included safety, costs, compliance with the Americans with Disabilities 
Act (ADA) and the use of commonly available materials. 

The playground problem brief differs from the pedestrian crossing in that the pedestrian flow problem 
provides no guidance on constraints and criteria which allows students to identify and specify their design 
parameters, potentially expressing why their unique solution is optimal. An example pedestrian flow 
problem looked like this, though details varied across schools to situated the problem in the local context: 

You are a team of engineers contacted by [your school name here] School District. Often 
hallways are congested at passing time between classes. Hallway one, which is between the new 
and old portions of the school, is difficult to navigate, [your school name here] School District 
would like your team to develop and propose a solution. 

Information gathering efforts were noticeably different across the two problems. Teams working on 
the pedestrian flow problem spent about twice the percentage of time searching for and digesting 
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information relative to the problem than did the teams working on the less familiar playground problem. 
Examples of information gathered in the playground problem focused on identifying typical components 
on playgrounds such as swings, slides, monkey bars and material characteristics such as strength, 
durability and cost. Information across both problems included benchmarking, but on different conceptual 
levels. Searching for playground components was a concrete task resulting in a list of typical play things 
while the hallway problem yielded much more complex transfer from other schools or public places 
where traffic congestion was a problem. Students looked at airports as examples of moving people in 
short periods of time as a potential method of benchmarking and gathered these examples to spawn ideas 
in their scenario. The transfer from an airport or mall hallway to a school hallway was challenging for 
students perhaps because of the population of users were different (i.e. adults in airports vs. students; 
adults may be motivated to run to their next flight vs. students who may not be interested in getting to the 
next class). 

Students in the hallway problem appeared to spend more time searching, perhaps motivated by their 
personal interest. In each administration of the problem, students were obviously bothered by the problem 
and were quick to engage as compared to the playground problem where students engaged at our request 
but seemed less intrinsically motivated. The hallway challenge included Internet access, which could have 
related to the additional search time. Our informal observations seemed to indicate that students not only 
accessed the computer, they also gathered information from memories of direct observations. They 
recalled their experiences in airports and malls with pedestrian congestion. They recalled traffic flow rates 
and locker placements in the school and considered the impacts of this information on their design 
process. 

Teams on the pedestrian flow problem spent more than twice the percentage of time considering 
feasibility of their solutions as compared to the teams on the playground problem. Feasibility was defined 
as considering the practicality or viability of a solution or element of the solution. This was differentiated 
from evaluation in that evaluation includes comparing two or more options while feasibility is passing 
judgment on one potential idea. In the playground problem, feasibility typically centered around cost in 
addressing the question: “Would an item/component cost too much?” Also, playground design teams 
considered the extent to which they met the constraints. This differed in the hallway problem because 
students were not provided with constraints or criteria nor did teams spend time to specify either 
constraints or criteria for the solution of the problem. Feasibility, however, consumed a much greater 
percentage of time as students attempted to determine if their solutions would work. Students implicitly 
must have identified some constraints and criteria as they talked about feasibility, but not directly. Typical 
examples included students discussing the financial cost of a solution or the impact that the proposed 
solution might have on the problem without explicitly identifying a budget or rationale that costs should 
be limited or minimal. Some solutions were structural while others were behavioral and students 
considered their potential solutions and students’ behavioral responses. This led to discussions of 
teachers’ roles, administrators’ media campaigns for pedestrian traffic patterns and the feasibility 
consideration: “would it work?” 

Feasibility seemed more relevant for students to consider in the hallway challenge as compared to the 
playground problem. Students were familiar with the hallway issues and the solutions had direct impact 
on their lives. They seemed to have capacity for understanding the complexities of hallway traffic more 
than complexities in the playground problem. Student design teams seldom considered the issues of safety 
in the playground, overlooking such facts as the difficulties that 2 year olds have in negotiating ladders. If 
the students had been parents of children for whom the playground was being designed, they might have 
considered safety and functionality with greater understanding, but as 17 and 18 year olds, they seemed to 
lack a sense of understanding about the functionality and dangers surrounding playground equipment 
design. 
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In both problems, evaluation and decision making activities were rarely observed. Student teams 
spent very little time comparing alternatives on a criterion, which was our working definition of 
evaluation. Students also spent very little time choosing among the alternatives. Decision making was 
defined in this study as a deliberate choice between two or more alternatives. Evaluation was defined as 
the comparison of two or more alternates using criteria. A typical decision and evaluation activity in the 
playground problem included material selection. Student teams would ponder using wood or metal as a 
construction material, discuss costs, strength and durability, then make a selection. In the hallway 
problem, very few evaluations or decisions were observed. This lack of evaluation and decision making 
may be directly related to the fact that students developed few alternative solutions during the 
brainstorming phase. While they did brainstorm and develop ideas, selection decisions tended to be 
related to the feasibility of individual components of the solution rather than a comparison of alternative 
solutions. Students would frequently say, “Let’s put in a slide, it’s cheap” with no externalized 
comparison of the alternatives. 

Teams were comprised of one gender or mixed genders. Four female teams, seven male teams and 
five mixed gender teams displayed differences in design processes. The gender of individuals assigned to 
the teams was difficult for the research team to control and, as a result, did not split equally across design 
problems. Three of the four female teams were concerned with the pedestrian flow problem while only 
one female team was concerned with the playground problem. Five of the seven male teams received the 
playground problem while two male teams received the pedestrian flow problem. Two of the five mixed 
gender teams engaged in the playground problem while three attempted the pedestrian flow problem. 

Results of team effort disaggregated by gender are presented in Table 4. Total design time varied 
across groups, with female teams finishing their work nearly 22 minutes before their male counterparts. 
Mixed groups of males and females averaged about eight minutes less than male teams. While female 
teams spent less time engaged in design, they spent more time on problem definition (8.4 minutes) than 
did the male teams (7.0 minutes) and mixed gender groups (5.2 minutes). This finding is particularly 
noteworthy, as most of the female teams were engaged in the pedestrian flow problem which, according 
to table 2, typically received relatively little attention to problem definition (4. 1 minutes). 


Table 4 





Mean and Standard Deviation Summary Statistics for High School Student Teams by 
Gender Composition 

Design Process 
Measures 

Female Only 
(n=4) 

Male Only 
(n=7) " 


Mixed Gender 
(n=5) 


, .. Percen 

Minutes 

(SD) of 

time 

Minutes 

(SD) 

Percen 
t of 
time 

, .. Percen 

Minutes 

(SD) ' of 

time 


Total Time 

87.1 

(13.3) 


108.8 (7.9) 


101.9(25.8) 


Problem Scoping stage 

26.6 

29.9 

27.1 

24.6 

27.9 

26.3 

Problem Definition 

8.4 (6.4) 

9.0 

7.0 (5.4) 

6.4 

5. 2(4.2) 

5.3 

Gathering 

Information 

18.2 

(14.9) 

20.9 

20.0 (9.7) 

18.2 

22.7 (12.9) 

21.1 

Developing Alternative 
Solutions stage 

44.5 

50.8 

58.4 

54.0 

59.9 

60.2 

Generating Ideas 

2. 4(0. 6) 

2.7 

2.9 (1.7) 

2.6 

3.1 (2.0) 

3.4 
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Modeling 

35.6 

(13.8) 


40.6 

Feasibility Analysis 

6.3 

(1.8) 

7.4 

Evaluation 

0.2 

(0.3) 

0.2 

Project Realization 

16.5 


19.6 

stage 


Decision 

0.2 

(0.3) 

0.2 

Communication 

16.3 

(3.8) 

19.4 

Other 

9.8 


11.3 


47.2(11.4) 

43.7 

46.2(12.8) 

45.3 

7.8 (4.4) 

7.2 

9.8 (5.2) 

10.7 

0.6 (0.8) 

0.5 

0.8 (0.9) 

0.8 

29.6 

27.3 

21.9 

20.9 

0.2 (0.3) 

0.2 

0.3 (0.4) 

0.3 

29.4(12.7) 

27.1 

21.5 (14.0) 

20.6 

10.2 

9.4 

6.7 

7.0 


Modeling and communication efforts account for most of the differences in overall design time. 
Females spent about 25% less time modeling than did males and mixed gender groups. The average 
female modeling time was 36 minutes while males and mixed groups spent 47 and 46 minutes 
respectively. Male student teams spent nearly twice as much time communicating (29 minutes) as did 
female student teams (16 minutes). 


Discussion and Implications 

Inter -rater reliability in modeling and feasibility were low despite extensive calibration efforts by the 
research team. Some of the lack of agreement could be related to researcher calibration, but the research 
team suspects that the lack of agreement is also related to lack of clarity by the student teams about the 
nature of these activities. The students were vague about how modeling was related to other aspects of the 
design process, particularly communication. What initially appeared to be graphical sketching as a 
method of developing ideas and laying out a potential solution for discussion evolved into a document for 
the final proposed solution. The boundaries between thinking on paper and communicating with external 
stakeholders blurred and presented the research team with difficulty identifying student intentions. At the 
end of the design challenge, students frequently presented rough sketches and messy notes, resulting in 
poor technical communication. If this situation is to be improved, it is important to make clear how to 
communicate technical information in a persuasive way to external stakeholders. Preparatory experiences 
might be focused on presentation skills where teams present their work to other classes or an invited 
audience of people who are not familiar with the daily student design experiences. This external audience 
would challenge teams to provide details and rationale for decisions made in context which was generally 
absent from participant work in this research. 

Other research efforts using student observations might increase their inter -rater reliability measures 
with teams by identifying what papers students are using when they are writing or sketching. In previous 
work with individuals, reviewing the digitized artifacts and video observation data typically provided 
ample evidence to determine what students were writing. However, with teams of students and multiple 
artifacts, researchers were less able to identify which paper was being used during a particular phase of 
the design process. Cameras positioned from an angle overhead might allow association between papers 
and content of the writing. However, in this research effort, a wide angle video of four students all with 
papers made identifying what was written and when difficult. In addition, the research team noticed that 
they were able to code feasibility, for example, consistently, but they had difficulty determining exact 
stai't and stop times. One research assistant might include a background statement as leading to feasibility 
while another might code a narrower band of feasibility leading to general agreement between researches, 
but low Kappa values. 


12 



Students engaged in dialog about the feasibility of their potential solutions or elements of solutions in 
the hallway problem more than in the playground problem, but problem definition was considered more 
in the playground problem. These two activities may be correlated such that a general lack of problem 
definition would lead to a tendency for students to be quick to question whether a potential solution 
would work. In the hallway problem, student teams seldom made explicit the constraints and criteria 
which made determining the feasibility of an idea more difficult and time consuming. The National 
Center for Engineering and Technology Education suggested, “As designs are considered for viability, 
optimization is essential. Students should make their value structures and goals for design success explicit 
early in the decision process. This sense of clarity provides opportunities to select and promote designs 
that make the most successful balance of trade-offs” (Householder & Hailey, 2012, p. 26). In the less 
structured hallway problem, students would consider an idea and then ask if it would work or be too 
expensive without having specified the definition of success or budget. 

Teachers should encourage students to identify the constraints and criteria as well as how success 
should be measured early in the design process. Feasibility considerations were slightly higher for team 
based design problems than individual design problems. Further research might test for a causal 
relationship, which if present, would indicate that teamwork might facilitate experience and exposure to 
critical thinking about solutions in the feasibility phase of the design experience. The National Center for 
Engineering and Technology Education caucus of 2012 suggested that “In collective team efforts, 
students may hold each other accountable for meeting criteria” (Householder & Hailey, 2012, p. 18). The 
sense of accountability may have been manifested in feasibility as students questioned each other’s ideas 
prompting consideration of flaws and opening the door for improvements. Stakeholder interests were 
included in student discussions of feasibility in the hallway problem much more frequently than the 
playground problem. In the hallway problem, they mentioned considerations such as how students would 
interact with their solution, how teachers would be involved and react to students in a redesigned hallway. 
They considered impacts of hallway reconstruction on the neighboring rooms and how the changes 
impact roles of librarians, cafeteria staff, and classroom teachers. Occasionally, parents and shopkeepers 
were mentioned in the playground problem as stakeholders, but with far less emphasis. This may be 
related to the sense of relevance provided by the hallway problem, because students cared in a very 
personal way about the success of their design and considered a larger system of stakeholders. 

Teachers may want to be sensitive to the gender composition of their student teams in view of the fact 
that single gender teams spent more time defining the problem than did mixed gender teams. This might 
suggest that encouraging and prompting students to develop an understanding of the problem is more 
urgent with mixed teams than with single gender teams. 

The general lack of evaluation and decision making may relate to a lack of alternatives for 
consideration. Students tended to think about new ideas until they had a few viable options and developed 
those into their final design. The lack of alternatives generally reduced the need to evaluate differences 
between them and reduced the number of decisions (choices between alternatives) to make. Teachers 
should encourage students to develop a significant list of alternative ideas before evaluation and decision 
making. Decisions regarding materials were made on the playground problem, but in both design 
problems, students did not develop many alternative designs. They considered the advantages of different 
materials but seldom considered holistically different solutions. 

Future Research 

Findings from this study suggest potential trends and correlations between individual design activities 
and group design work as well as suggesting differences between teamwork on two different kinds of 
problems. This study had a sample size of 17 teams in total. Eight teams were challenged with the 
playgroup problem and nine teams engaged in the hallway problem. With only eight or nine teams in 
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comparison, statistical analysis was not conducted. This work is potentially foundational to larger studies 
as it may allude to trends and correlations that could be tested in experimental or quasi-experimental 
research conditions on a larger scale. 

From a methodological perspective, the inverse relationship between time spent in problem definition 
and time spent in feasibility might provide insight into student problem definition. Though problem 
definition in the hallway design task was seldom coded, future researchers could use feasibility as a 
method of extracting student definitions of the problem. Feasibility considerations have inherent value 
statements that could provide a proxy for problem definition. Therefore, by analyzing student 
conversation about the feasibility of an idea, a future research team may be able to identify the implicit 
constraints and criteria that students do not mention explicitly. For example, if a student judges a potential 
solution to be too expensive, we can infer that cost is a criterion even though it was not mentioned as one. 
Students mentioned concerns such as slide or platform height or soccer field location in the playground 
problem, which relate to the constraint presented that the playground must be “safe”. In this example, the 
students silently operationalized safety by considering a minimum distance from a nearby road to the 
soccer field. In addition, students’ brainstorming activities may provide insight to the problem definition 
in that they tend to think of potential solutions and the commonalities across those solutions may be hints 
into their problem definitions. As an example, students who list different ways of controlling student 
hallway traffic such as traffic lights, teachers, or mirrors, may suggest implicitly that widening the 
hallway is not practical or that they feel constrained by the lack of resources to make major structural 
changes in the school architecture. 

Further research might investigate qualitative differences in the ways and methods in which all 
female teams engaged in problem definition as compared to males and mixed groups. Female groups 
spent more time and a much higher percentage of time on problem definition than their peer groups. 

Additional investigation might clarify differences between genders in terms of modeling and 
communication activities. Female groups spent dramatically less time modeling and communicating 
which accounted for most of the differences in overall design problem time. Are females modeling and 
communicating less or differently than their male and mixed gender teams? Would this effect be reduced 
if more explicit modeling and communication expectations were provided to the teams? 

Summary 

In this study, seventeen teams, each comprised of 2-4 high school students, were asked to complete a 
team based engineering design challenge. Observational protocol analysis was conducted based on a 
foundation of previous work, including the adoption of previous coding schemes. Differences between 
groups and individuals were compared. Teams of students were split in two groups; one set of teams 
received a playground design problem while the other received a hallway design problem. Teams worked 
up to two hours after school on the design problems and provided the recommendations resulting from 
their work at the conclusion of the session. 


This material is based upon work supported by the National Science Foundation under Grant No. 0426421. Any 
opinions, findings, and conclusions of recommendations expressed in this material are those of the authors and do 
not necessarily reflect the views of the National Science Foundation. 
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